Three-dimensional face identification system

ABSTRACT

A three-dimensional (3-D) facial imaging system is disclosed for generating facial images, indexing those images by composite codes and for searching for similar two-dimensional (2-D) facial images. The 3-D images of human faces are generated from a data repository of 3-D facial feature surface shapes. These shapes are organized by facial feature parts. By assembling a shape for each facial part is assembled, a 3-D facial image is formed. Human facial images may be represented with a composite code (facial feature part code and shape code). A 3-D image of any face may be generated using the code to select the proper facial feature shapes from the stored repository of shapes.

Applicant's provisional application entitled “Integrated Law-EnforcementFace-Identification System”, U.S. Ser. No. 60/067,065 and filed Dec. 1,1997, is a related application and is incorporated by reference in itsentirety.

FIELD OF THE INVENTION

This invention relates to the field of face recognition, and, inparticular, to face-identification for law-enforcement and correctionsapplications.

BACKGROUND OF THE INVENTION

Law enforcement organizations rely heavily on witnesses to a crime forpurposes of identifying criminals based on pictures of the criminal.Typically, a witness will review pages and pages of “mug shots” ofcriminals from books of mug shots maintained by the law enforcementorganization. The witness hopefully recognizes the criminal from the mugshots.

Over the years, huge collections of mug shots have been developed bylocal, state and federal law enforcement agencies. Some of thesecollections of mug shots have been deposited in a national depositoryfor mug shots maintained by the Federal Bureau of Investigation (FBI) ofthe U.S. Department of Justice. It is estimated that in the UnitedStates there are approximately 60 million mug shots in various mug shotcollections across the country. However, the FBI has only aboutone-third of all U.S. mug shots in its own collection.

Mug shots are typically a frontal and profile views of an individual. Amug shot usually comprises two photographs of the front and side view ofthe face of an individual. To provide standardization of mug shots, lawenforcement photographers follow specific guidelines on the positioningof the front view and side views of a person for photographing mugshots. Accordingly, each pair of mug shots, i.e., the front face andside face views, is uniform with respect to angle of view, distance andlighting of the person being photographed. These mug shots have beenstored in photographic books, on film, on tape and now digitally oncomputer discs.

Mug shots show only two views of an individual—the front face view and aprofile face view. To recognize an individual suspect from mug shots, awitness must recognize the front face or profile view of the individual.However, the witness may not have seen the suspect from the front orprofile view. Rather, the witness may have seen an angled view of thesuspect's face, such as from above, below or to one side. The view thatthe witness sees at a crime scene is often different than the front orprofile view shown in mug shots. For a witness to recognize a suspect byreviewing the mug shots, the witness must convert in his own mind theangle from which he saw the suspect to either or both a front face orside view of the individual as shown in the mug shots. Not all witnessescan reliably convert in their mind the view actually seen of the suspectto the front face or side views shown in mug shots. Because allwitnesses cannot make this conversion, witnesses from time to time havenot been able to identify suspects. Moreover, witnesses at criminaltrials are subject to cross-examination about how they recognized asuspect based on the frontal or profile view shown in a mug shot whenthe witness never saw the suspect from those particular views.Accordingly, criminals have not been identified or have been acquittedat trial because the witness did not reliably recognize the criminalsuspect when reviewing mug shots, or because the witness was effectivelycross-examined at trial with respect to the view that he had of thesuspect.

There are many other facial features that are more difficult to describeand identify, such as the size of a person's nose, description of theirface, etc. These other facial features are often recalled and describedby witnesses to a crime, and can be used to recognize a suspect.However, these other facial features are not easily categorized and,thus, have prevented the creation of an effective system forcategorizing or indexing mug shots. In the past, law enforcementinvestigators have attempted to characterize facial features. Forexample, law enforcement organizations have created standard measurementtools for identifying the size of person's eyes, nose, mouth, facialstructure and ears. The criteria that are used are primarily used foridentifying mug shots are limited 2-D criteria based on a full frontalview or profile view of a face. For example, the criteria as to nosewidth and eye shape may be limited to just front view figures, and notto profile views. Similarly, ear shape criteria may be applicable onlyto profile views and not to frontal view pictures of the individual.Thus, these criteria are of limited use at best if the witness observedan individual from a perspective other than a frontal or profileperspective.

Prior to the present invention, no known techniques were used forindexing facial features based on three-dimensional (3-D) framework.Prior approaches to cataloging and indexing facial features have assumedthat the faces are shown only in 2-D full frontal and/or profiled viewssuch as in mug shots. These indices of profile and frontal views of mugshots are helpful in identifying suspects when a front face or profileface view is available of the suspect. For example, if the suspect looksstraight into a security camera such that there is a front face image,then that front face image can be indexed using standard face indices.Using the indices of the security camera picture of the suspect, theindicie values can be used to identify mug shot pictures having the sameindicie values in searching for the criminal. Similarly, if a witnessworking with a law enforcement sketch artist can generate a frontal faceand/or profile view of a suspect, that frontal face and profile sketchcan be used to categorize the facial values of the criminal. Thesefacial values can also be used to access the mug shots which are by thefacial values indices used to obtain those mug shots having the samefacial value indices.

The indices of facial features used to index mug shots are not usefulwhen a frontal face or profile picture or sketch of the perpetrator isunavailable. Side views of a suspect cannot be readily categorized byfacial indexing used with the current mug shots. A side view picture ofa criminal is not susceptible to identifying the facial feature valuesused with current indices of mug shots. Unfortunately, the majority ofphotographs taken of suspects by security cameras are of angled views ofthe suspect's face other than frontal face or profile. Thus, when asecurity camera captures on film a suspect, often that picture cannot bedirectly used to identify the suspect by mug shots.

It has been long recognized to be an undesirable trait that the mug shotindices do not allow for correlation with pictures of criminals otherthan straight front or straight profile views. It is a long-felt need inlaw enforcement to have a computerized system for recognizing faces inwhich a photograph or other 2-D image of a criminal from any view can bequickly and reliably analyzed and compared to a mug shot collection toselect those mug shots of possible perpetrators of a crime.

SUMMARY OF THE INVENTION

The present invention is a three-dimensional (3-D) face-identificationsystem useful for human faces and other image identificationapplications. The invention also provides a fast search engine forsearching a large collection of two-dimensional mug shot photographs forthose that match a particular facial image. In addition, the inventionprovides a technique for indexing 2-D mug shots to a uniform set of 3-Dfacial feature parts, such that a 3-D face surface images are generatedbased on the 2-D mug shots. Moreover, the invention provides a method bywhich a large number of mug shots may be reproduced using 3-D facesurfaces by reference only to the uniform set of 3-D facial surfacefeatures and a composite code that identifies the facial feature partsto be used to generate a particular individual's facial image. Thismethod avoids the need to locally store a large amount of data thatwould be required if every mug shot were stored locally. In addition,the invention has the advantage of generating a 3-D facial image fromany desired point of view, and of providing a searchable code to 2-D mugshot photographs from a 3-D image reconstruction of a face.

The present invention provides three-dimensional face-identificationsystem which incorporates facial fiducial values (points). Thesefiducial values are used to index a facial image determining thecomposite code for the image. The composite code is used to regeneratethe facial image from a group of standard facial feature parts. Thepresent invention allows facial fiducial values to be measured from anyphotograph, sketch or picture of an individual's face from anyviewpoint. For example, a photograph of the face of a suspect from theperspective of 45° off to the side of the suspect. The photograph isanalyzed to determine its facial fiducial values corresponding to athree-dimensional (3-D) model of the face. Once these facial fiducialvalues are obtained with respect to a three-dimensional model, they areused to determine the proper composite codes for the 3-D facial image.These composite codes correspond to a facial feature parts database.Thus, the suspect's 3-D face can now be generated using only thecomposite code and facial feature parts. In addition, the compositecodes correspond to composite codes that index the two-dimensional mugshot photos. Thus, existing mug shot directories can be quickly searchedby finding mug shots that have similar composite codes to the compositecode for the face of the suspect to select mug shots of possibleperpetrators. The selected mug shots are then accessed, preferably by acomputer system and displayed on a computer screen to the witness.

In addition, the invention is able to convert the two-dimensional mugshot photos to a three-dimensional image. This three-dimensional image(shown as a 2-D display image), can be rotated automatically or manuallyby the viewer to a viewpoint that corresponds to the view that thewitness had of the suspect. Moreover, the facial image shown on adisplay can be rotated to the viewpoint corresponding to the viewpointof a suspect taken by a security camera or other photographic or videodevice. In this manner, more reliable and faster identification ofsuspects can be accomplished using standard two-dimensional mug shotsthat are selected using three-dimensional imaging fiducial featurepoints and composite codes. The face-recognition technology of theinvention is based on a completely three-dimensional (3-D) framework.This critically-important approach of the invention is conceptuallydifferent from all existing face-recognition technologies. The inventioncompletely adheres to the fact that naturally 3-D surfaces, whichinclude 3-D face (head) surfaces, cannot be compared in differentangled-view pauses based on two-dimensional (2-D) planar images,including the 2-D face images as frontal-view and profile-view mugshots.

Therefore, the invention establishes its 3-D framework based on 3-D face(head) surfaces 102, 104 (FIG. 1), which are subsequently used toestablish the face-feature surfaces of the face-feature parts. Theinvention uses the available laser-scanner data for 3-D head surfaces(e.g., data generated by CARD Laboratory of Wright Patterson Air ForceBase, U.S. Air Force). The available data for the 3-D head surfaces isused to generate the required 3-D surface mesh (grid/panel) structures202 which can be considered in high-resolution 204 (FIG. 2),intermediate resolution 302 (FIG. 3) and low-resolution 402 (FIG. 4)formats for rendering 3-D twenty-four (24)-bit color image of facial andhead surfaces.

The invention uses data for 3-D head surfaces (i.e., for more than 2000humans) to establish a face-feature-surface repository. The inventionidentifies on each available 3-D head surface, up to thirty-two (32)face-feature parts 502 and separates their associated 3-D face (head)feature surfaces 504. The invention associates an “index i” to each face(head) feature part 502 (e.g., nose as i=1, upper-lip as i−2, lower-lipas i=3, chin as i=4, etc.) to establish the required computerbookkeeping or indexing procedure. The invention considers the 3-Dhead-surface data for each different head; and by separating the face(head)-feature parts 502 for each head surface image, a computergenerates a complete set of different face (head) feature surfaces 504to form a repositgory of face feature surfaces.

The invention considers up to two-hundred fifty-six (256) differentface-feature surfaces for each one of the designated face-feature part(i). Therefore, for each of the designated face-feature parts “i” (e.g.,nose, upper lip, lower lip, chin, left eye, right eye, etc.), theinvention establishes up to 256 face-feature surfaces for each part byusing the available laser-scanned 3-D head-surface data of up to 2000heads. Conventional geometrical clustering techniques are used tocategorize up to 256 distinct face-feature surfaces 504 for eachdesignated face-feature part “i”, from up to 2000 face-feature surfacesobtained from the available laser-scanned 3-D head-surface data.

To each categorized, distinctly-different face-feature surface 504 ofthe designated face-feature 502 (part i), the invention assigns aface-feature-surface code “j”, which is associated with all of its knowngeometrical and surface-color/complexion characteristics. Therefore, forcomputer bookkeeping purposes, the invention establishes a completely3-D framework, based on a set of “indices i”, which identify thedesignated face-feature parts, and subsets of “codes j”, for theface-feature surfaces, such that for any specified face-feature index iand the face-feature-surface code j, the invention can provide acompletely 3-D, color face-feature surface, which can be renderedgraphically in the frontal view, left-profile view, right-profile viewand in all possible angled-view for observation and analysis.

Based on the results of its geometrical clustering analysis, theinvention considers sets of face-feature-surface codes j, which containlow-numbered codes, starting with one (1) for the most commonface-feature surfaces, and high-numbered codes, ending with two-hundredfifty-five (255), for the least-common (generally most prominent)face-feature surfaces. Fewer than 256 face-feature surfaces may be usedfor each face-feature part. For example, sixteen (16) face featuresurfaces may be used for each face-feature part.

The invention considers, currently sixty-four (64), or as required up toone-hundred twenty-eight (128) face attributes, which include thedesignated, up to 32, face-feature parts. FIGS. 5 and 6 show someexemplary face and head feature parts 502. The invention uses theadditional face attributes for considering such features as: (a)additional geometrical information (e.g., relative distances between theface-feature surfaces) of the face-feature parts; (b) color, tone,complexion, texture conditions of the face surfaces; (c) facial hair; d)abnormal characteristics as scars, bone damage, etc., and (e) otherfeatures.

The invention maintains, for each face-feature surface (with code j) ofeach face-feature part (with index i), a 3-D high-resolution (FIG. 2)surface mesh (vertex/panel structure) with 24-bit color information(FIG. 8) for graphically rendering 3-D face (head) surfaces. In order tofurther simplify the geometrical comparisons of different face-featuresurfaces, the invention considers, for each face-feature part, a set ofselected fudicial points which represent readily-identifiablecharacteristics locations on its face-feature surface. As an example,for the nose part, as the face-feature part with index i one (1), theinvention considers twenty-one (21) points, including the top point,bottom point, tip point, nostril-side points, top-flare points, halfpoint, quarter points, etc. (FIG. (9). The sets of fiducial pointscollectively guarantee that two nose-surface with matching sets offiducial points, also match geometrically, within the limits of humanperception and machine intelligence. As another example, for the earparts, i.e., left ear, with index i as thirteen (13) and right ear withindex i as fourteen (14), it considers eight (8) points, including thetragion points, maximum-back point, maximum-outward point, etc. (FIG.10), which allows the invention to incorporate thegeometrically-important characteristics of the ears in its 3-Dface-recognition technology.

The invention considers the use of fiducial points with two novelconceptual and implementation features, that are uniquely different fromall existing 2-D face-recognition technologies. First, the inventionconsiders the sets of fiducial points, in the sense as fiducialtemplates, for all the face-feature parts, in contrast to the whole 2-Dface image (or slightly distorted 2-D face images), associated withconventional 2-D face-recognition technologies. Second, the inventionconsiders completely 3-D positions for the fiducial points, in contrastto the 2-D positions used in conventional 2-D face-recognitiontechnologies. Therefore, the invention, by considering the completely3-D structure of the sets of fiducial points (i.e., fiducial template)can generate correctly all the 2-D structures of the fiducial points onplanes normal to the direction of line of sight to the face (head) forall angled-view observations.

The invention considers up to 256 3-D face-feature surfaces, identifiedby their “codes j” or each of the 64 face-feature parts, designated byits “index i”. For each face-feature part, e.g., nose with i as 1 (FIG.11), chin, with i as 4 (FIG. 12), left/right ears, with i as 13 and 14,respectively (FIG. 13), the 3-D framework of the invention provides allthe associated 2-D images of the 3-D face-feature surface as observedfrom any specified view, including the frontal view, left/right-profileview, and any left/right- and upward/downward-angled view. Therefore,the 3-D framework of the invention, which is based on 3-D face(head)-feature surfaces, can completely eliminate the geometricalproblems associated with angled-view observations, as encountered incurrently-available 2-D face-recognition technologies.

The invention uses its 3-D framework, based on face-feature parts (e.g.,nose, upper lip, lower lip, chin, left eye, right eye, etc.), with eachface-feature part that can be represented by one of the up to 256associated face-feature surfaces (FIGS. 5-10), to construct distinctlydifferent 3-D face-composites which can be rendered, in 24-bit color, as3-D face (head) surfaces. According to the invention, the bookkeepingrequired for constructing a 3-D face (head)-composite consists of up tosixty-four (64), face-feature parts, identified by their “indices i” andthe associated 3-D face (head)-feature surface, identified by its “codej” (FIG. 14). The critically-important consequence of the 3-D frameworkof the invention, is that, with the already-stored 3-D, 24-bit colorgraphical information for all the face (head)-feature surfaces (up to256) for each of the face (head)-feature part (up to 32), the systemonly requires information as “code j” integers, between 1-255, for each“index i”, to be able to render in 3-D, 24-bit color format the completeface (head)-composite. Considering that each “code j”, associated with aface (head)-feature surface represents an 8-bit, i.e., 1 Byte integernumber (binary), and considering up to 64 (or 128) face-featureattributes, including the face (head)-feature surfaces (up to 32), theinvention require the specification of a maximum of 64 (or 128) 1 Byteintegers, with a maximum storage requirement of only 64 Bytes (or 128Bytes) for each composite-face (head) code (FIG. 14). Therefore, theinvention provides the capability to store up to 60 million mug shotsets, based on frontal view and profile-view 2-D face images, with lessthan 4.0 Gigabyte of storage.

The mug shot conversions functionality of the invention provides theprocedure for converting a set of two mug shots, as 2-D face images,into a 3-D face (head)-composite (FIGS. 15 through 25). First, theprocedure considers the appropriately-cropped and sized 2-D face imageof the frontal-view mug shot. It initiates the matching, e.g.,conversion procedure, by finding among the already-stored nose-partrepository (“index i” as 1) of the nose surfaces, and installing on abase head, the nose surface (e.g., “code j” as 090) that most-closelyresembles the nose on the 2-D frontal-view face-image (FIG. 15). Itaccomplishes the appropriate matching based on geometrical comparisonsof the 2-D locations of the fiducial-point templates of the 3-D nosesurfaces (FIG. 16).

Second, the procedure considers the appropriately-cropped and sized 2-Dface image of the profile-view mug shot. It continues the conversionprocedure by matching, independently, and installing on the base head,the nose surface (“code j” as 090) which most-closely resembles the noseon the 2-0 profile-view face image (FIG. 17). It accomplishes theappropriate matching based on the geometrical comparison of the 2-Dlocations of the fiducial-point templates of the 3-D nose surfaces (FIG.18). Third, it considers the results of the two independent matchingprocedures and decides, based on minimum geometrical errorconsiderations, the 3-D nose surface (“code j” as 090) whichmost-closely resembles the nose in both 2-D frontal-view andprofile-view face images (FIGS. 16 and 18).

The mug shot conversion functionality of the invention continues thematching procedure for all, up to 32, facial-feature parts, including,e.g., the left/right-ear parts (FIGS. 19 and 20), and systematicallycontinues the assembly of the face (head)-composite, by considering boththe frontal-view and profile-view observation of the partially-assembledface (head)-composites at different levels (FIGS. 21 and 22). Finally,for the two, i.e., frontal-view and profile-view, 2-D face images of themug shot set, the mug shot-conversion procedure of the inventionestablishes the completely assembled 3-D face (head) composite (FIGS. 23and 24).

Two consequences of the mug shot-conversion procedure of the inventionis critically important. First, the final, completely-assembled 3-D face(head)-composite naturally provides geometrically accurate and realisticrepresentations of the frontal-view and profile-view observationscorresponding to the set of two mug shots. But more importantly, it canalso provide geometrically accurate and realistic representations of anyand all specified angled-view observations of the 3-D face (head)composite corresponding to the set of two mug shots. Therefore, first,the invention establishes a geometrically-accurate and realistic 3-Dface (head)-composite corresponding to the original 2-D face images ofthe mug shot set, based on, importantly, appropriate assembly of the 3-Dface surfaces (with “code j”) of the face-feature parts (with “indexi”), which are maintained in its face-feature-part repositories. Second,the invention establishes the essential face (head) composite code whichrequires at most 64 Bytes.

The mug shot conversion functionality accomplishes the conversion of theavailable 2-D, pixel-intensity-dependent face-image data, associatedwith the original set of two, frontal-view and profile-view mug shots,with a face (head)-composite code of at most 64 Bytes (FIG. 14).Consequently, the invention provides the ultimate data-compressionformat for storing the required information for the mug shot records.

The general picture-conversion functionality of the invention providesthe procedure for converting a single, angled-view (arbitrary) 2-D faceimage of a person, generally as an unknown person, into a 3-D face(head)-composite (FIGS. 27 through 31). The picture conversionfunctionality of the invention is based on the same procedure of the mugshot conversion functionality, with the exception that the procedure isnot limited to considering only the frontal-view and profile-view 2-Dface-images of the mug shots, but one or more angled-view 2-D faceimages obtained as captured pictures (generally obtained fromsurveillance equipment). The picture conversion procedure considers theappropriately-cropped and sized 2-D face image of the availableangled-view picture. It initiates the conversion procedure by matching,i.e., finding among the already-stored nose-part repository (“index i”as 1) of the nose surfaces, and installing on an appropriately-orientedbase head, the nose surface (e.g., “code j” as 090) that most-closelyresembles the nose on the 2-D angled-view face-image (FIG. 27). Again,it accomplishes the appropriate matching based on geometricalcomparisons of the 2-D locations of the fiducial-point templates of the3-D nose surfaces (FIG. 28). An important aspect of the pictureconversion functionality of the invention is that the proceduresystematically considers different angled-view conditions of thefiducial points of the 3-D nose-template to match simultaneously boththe angled-view conditions (e.g., 20 degree rotated, 15 degree upward,and 5 degree sidewise tilted) and the 3-D nose surface (e.g., “code j”as 090).

Again, the picture-conversion functionality of the invention continuesthe matching procedure for all, up to 32, facial-feature parts,including, e.g., right-ear part (FIG. 29) and systematically continuesthe assembly of the face (head)-composite, by considering only thesingle partially assembled face (head)-composites at different levels(FIGS. 30 and 31). Finally, for the given 2-D angled-view face-image asthe captured picture of a person, the picture-conversion procedure ofthe invention establishes the completely assembled 3-D face(head)-composite (FIG. 31).

The two consequences of the picture-conversion procedure of theinvention are critically important. First, the final,completely-assembled 3-D face (head)-composite naturally providesgeometrically accurate and realistic representation of the givenangled-view observation of the picture of the person. Again, moreimportantly, it can also provide geometrically accurate and realisticrepresentations of any and all specified angle-view observations of the3-D face (head)-composite, corresponding to the given 2-D angled-viewface-image of the picture.

Therefore, again, the invention establishes a geometrically accurate andrealistic 3-D face (head)-composite corresponding to the original 2-Dface-image of the picture, based on, importantly, appropriate assemblyof the 3-D face-surfaces (with “code j”) of the face-feature parts (with“index i”) which are maintained in its face-feature-part repositories.Second, again, the invention establishes the essential face(head)-composite code which requires at most 64 Bytes of binary datastorage (FIG. 14).

The invention also provides a novel 3-D composite-generator system (FIG.32) which can be used to generate 3-D composites of perpetrators (orsuspects) based on information provided by the victim(s) and/orwitness(es). The completely 3-D composite-generation system is designedand constructed as a stand-alone utility (user interface) which cangenerate 3-D face (head)-composites under all angled-view observationand special lighting conditions. Therefore, the composite generatorsystem (FIG. 32) of the invention allows the law-enforcement/correctionspersonnel to generate realistic 3-D face (head)-composite according tothe conditions similar to the ones encountered by the victim(s) and/orwitnesses. The 3-D composite-generator system automatically generatesthe essential face (head)-composite code which can be used on the flyfor comparisons of the generated 3-D face (head)-composite with theavailable (already processed) mug shot records.

Some of the novel features of the present invention include:

The 3-D face-recognition technology is compatible with existing 2-Dimages of mug shots. The 3-D images may be projected onto a display as2-D profile and frontal views of a 3-D face image. The 2-D images arealso mapped onto 3-D surface facial feature parts to convert the 2-Dimages into 3-D images. In addition, the indexing system used inconnection with the 3-D images is also applied to the existing 2-Dimages so that the 2-D images may be retrieved based on the index forthe 3-D images.

The 3-D face-identification technology, in which 2-D mug shotphotographs are mapped to 3-D facial feature parts, allows angled viewpictures of individuals to be generated from the existing 2-D mug shots.Accordingly, a witness may view a suspect from an angled view picture,and is not limited to the frontal and profiled views of a suspectprovided by standard mug shots.

The 3-D technology generates each 3-D human face or head from 3-D facialfeature parts, e.g., chin, eyes, nose, mouth, etc. In one embodiment ofthe invention, the human face and head has been divided into sixty-fourfacial feature parts. Each part is a 3-D facial surface image of ananatomically significant feature of a face or head. There may besixty-four (64) facial feature parts. When all 64 facial feature partsare assembled in a mosaic a complete 3-D human face is formed. For eachfacial feature part, a wide variety of different part shapes, (such as256 different facial feature surface shapes) are available and stored asa 3-D facial feature image surface. For example, a nose facial featurepart may have 256 different nose shapes that can be selected ingenerating a particular nose for a facial image. Virtually any humanface can be generated from the 64 different facial feature parts, andthe group of facial feature shapes for each part.

Each shape in a group of a facial feature part is ranked in order of itsdistinctiveness with respect to the other shapes in the same group. Forexample, the most distinctive noses, e.g., large protruding noses,having a high ranking, and small, non-descript noses having a lowranking. Highly distinctive facial features tend to be the most quicklyrecognized and the most long remembered by witnesses. By assigning ahigh ranking to distinctive facial feature parts, these quicklyrecognized and most long remembered are displayed first to speed thefacial recognition process.

Each of the facial-features parts and their associated shapes are storedin local memory as high-spatial-resolution-3-D geometric surface shapes.There may also be color complexion data that is locally stored for usein applying skin color tones to the generated 3-D images of faces andheads. The amount of memory required to store the data for all of the3-D 64 facial feature parts and the associated 256 part shapes is abouttwo gigabytes of binary storage space. In contrast, the memory spacerequired for the available forty million 2-D mug shots would requireabout 73,780 gigabytes, and even if these mug shots are compressed thememory requirement would be 734 gigabytes. It is substantially morepractical to locally store on a law enforcement computer 3-D facialfeature parts that can be assembled to form any face image, rather thanto locally store the actual 2-D images.

With the present invention, the number of facial images is not adominant factor determining the memory storage requirements. All humanfaces can be formed from the facial feature parts stored locally to alaw enforcement computer. An image of any particular individual may begenerated from the facial feature parts stored in the computer. Thereare identifying codes (i, j composite codes discussed below) for thoseindividual's whose faces have been cataloged into a mug shot database.The identifying codes are used to identify the features of theindividual's face. Moreover, the identifying codes are correlated to the3-D facial feature parts. The memory required to store the identifyingcodes of an individual is negligible as compared to the memory spacerequired to store an actual image of the individual. The presentinvention can generate a facial image of an individual by selecting thefacial feature part shapes that match the individual based on theidentifying composite code for the individual. Accordingly, anincreasingly larger number of identifying “composite” codes can bestored locally, without requiring substantial additional memory.

The present invention also has the ability to recognize a suspect'sfacial image by assigning identifying composite code values to theimage, and then using these codes to select mug shot photographs havingthe same and similar identifying code values. The shape of each facialfeature part is identifiable by a low spatial resolution 3-D techniquethat uses a relatively small number, (e.g., less than fifty (50) pointsand, in the preferred embodiment is only 32 points), facial featurefiducial points. These fiducial points are assigned to anatomicallycritical points of a facial feature part. A fiducial point is a point onthe surface of a facial feature part that can be used to identify theshape of the particular part and to assign a proper composite code tothe facial feature. Each fiducial point is specified by a positionvector that defines a position of the surface point. The position vectormay be defined by only thirty (30) bits of data. Each facial featureshape may be identified by 32 position vectors that locate the positionsof surface features of the shape of the facial part. The fiducial pointsare used to identify a facial image and to compare the image of one faceto other facial images. By identifying the fiducial points on thepicture of a suspect, the suspect's facial image may be mapped to the3-D facial feature parts and used to determine the composite codes forthe facial image and generate a 3-D facial image of the suspect. Inaddition, once the composite codes may be used to search for otherfacial images that are the same as or similar to the suspect's facialimage. Accordingly, the invention has the capability of facial imagerecognition using a relatively small number of facial feature fiducialpoints.

The invention also provides a fast search engine for finding facialimages that are the same or similar to the facial image of a suspect orother person. By mapping the fiducial points of a facial image of anindividual, the composite codes corresponding to the individual's facialimage can be determined. These composite codes are used to search forother facial images (in mug shot databases) that have the same orsimilar composite codes. The search for the same and similar compositecodes in tens of millions of mug shot codes is done much faster thanwould an image-to-image search through the mug shot images. Thus, thepresent invention allows for a quick search to be conducted of other mugshots that match the image of a suspect.

BRIEF DESCRIPTION OF THE DRAWINGS

These, as well as other objects and advantages of this invention, willbe more completely understood and appreciated by careful study of thefollowing more detailed description of a presently-preferred exemplaryembodiment of the invention taken in conjunction with the accompanyingdrawings, of which:

FIG. 1 is a diagram showing two angled views of a 3-D human head image;

FIGS. 2 to 4 are wire frame images of the head image shown in FIG. 1;

FIGS. 5 and 6 are illustrations showing exemplary facial-feature parts;

FIGS. 7 to 10 are illustrations showing various views of exemplary facesurface, including images with fiducial points;

FIGS. 11 to 13 illustrate composite codes and associated facial featureparts and surfaces;

FIG. 14 shows an exemplary list of composite code of a 3-D face and headimage;

FIGS. 15 to 18 are diagrams illustrating the adding of a nose facialsurface to a face image, and the fiducial points used to map a nose;

FIGS. 19 to 24 shows the fiducial points for an image of an ear and theaddition of an ear to a face image;

FIG. 25 shows various views of a partially-completed facial image;

FIG. 26 shows an exemplary composite code;

FIGS. 27 to 30 show left angled views illustrating the generation of afacial image;

FIG. 31 shows face images generated from composite codes and facialfeature surfaces;

FIG. 32 shows an exemplary screen display used in generating a facialimage;

FIG. 33 is a schematic diagram of a computer network capable of runningthe present invention, and

FIG. 34 is a flow chart showing an exemplary computer program forimplementing certain aspects of the present invention.

DETAILED DESCRIPTION OF THE DRAWINGS

FIG. 33 shows a schematic diagram of a computer network for operatingthe present invention. The invention may be operated on a wide areanetwork (WAN) 3200 or on separate computer systems that share data bysome means, such as modem, Internet, physical disk transfer or wirelessconnection. The network includes a communication network 3202 that maybe implemented via the Internet, direct modem-to-modem telephone links,wireless, satellite or other commonly-available network technologies forinterlinking computer systems. The communication network may link acentral mug shot depository computer 3204 to a plurality of local lawenforcement computer systems 3206. The wide area network allows lawenforcement officials from their local computer system to access mugshots stored either on their own computer system and/or on a centraldepository computer 3204. Moreover, the mug shots may be stored on otherlaw enforcement computer systems and accessed remotely by another lawenforcement computer.

The central mug shot depository computer 3204 may be a mug shotdepository computer system to be maintained by the United StatesDepartment of Justice, and, in particular, the Federal Bureau ofInvestigation (FBI). The FBI maintains a central mug shot depository ona computer which is accessible over modem lines by local law enforcementcomputers to obtain mug shot photographs. It is envisioned that the FBIcentral computer system will maintain a database 3208 of mug shot photos3210, uniform 3-D facial features images 3212, and mug shot deposit codedata 3214.

The data for the mug shot photos may be two-dimensional images of anindividual's face, showing the face in profile and plan views. Theimages may be stored in a standard image file format, e.g., JPEG, GIF,TIT, etc., which utilizes some form of data compression to save storagespace. The images may be in black and white, or color. The imagespreferably conform to normal formats for law enforcement mug shotsphotos, in terms of field of view, picture lighting, camera distance toindividual, etc.

The amount of memory space taken up by each mug shot image is relativelylarge. Considering that the FBI's mug shot database includes tens ofmillions of images, the amount of computer memory space required tostore all of these images is tremendous. The central database 3208 isthe primary storage facility for the mug shot data. It is practicallyimpossible for local law enforcement stations to store a large number ofmug shots as image files. Typically, local law enforcement store on itslocal computer memory 3220 a database 3222 of images of persons who havebeen suspects in the local region of the law enforcement organization.Thus, often times the local law enforcement organizations do not haveaccess to mug shots of individuals outside of their local areas. Thereis a need for local law enforcement organizations to have access to mugshots that are not stored in their local computer systems.

The invention provides a means to characterize individuals shown in mugshots by their 3-D facial features. In particular, a database 3212 ofuniform 3-D facial feature parts is maintained in the central database,and in local computer databases 3220. FIG. 1 shows examples of a 3-Dhuman head (right-angled view 102 and left-angled view 104) stored asgraphical wire frame images 202 (FIGS. 2 to 4) in computer memory. Arelatively-low resolution wire frame image 402 is shown as being formedof a set of parallel and perpendicular lines, a medium resolution wireframe image 302 and a high resolution wire frame image 204 is shown as aset of geometric lines more closely spaced together than in thelow-resolution image 404 or medium resolution image 302. Other graphicaltechniques may be used to depict 3-D images, such as a bit map image,character mapping, etc. that are well-known to persons skill in computergraphical imaging.

Each human head is divided into facial feature parts, as is shown inFIGS. 5 to 6. Each facial feature part is a section of the facialsurface feature 502 of the 3-D digital head images, e.g., a mouth, nose,eyes, forehead, chin, ears, etc. As is shown in FIGS. 7 to 13, eachgeneric facial-feature-part 502 is assigned an “i” number 702, which inthe case of a nose may be the numeral “01”. See FIGS. 14 and 26. This“i” number 1400 is used to identify the generic facial feature part,e.g., chin, mouth, eye, etc. The facial feature parts areanatomically-prominent facial features. In addition to features commonto all human faces, they may include facial defects, such as scars, andother features, such as tattoos.

In addition, each shape variation of a facial feature part is assigned a“j” number 704. FIG. 14 identifies a group 1402 of “j” codes for eachpart 1400. As shown in FIG. 12, a relatively square chin is assigned jnumeral 001, a protruding chin is assigned j numeral 082 and a roundedchin is assigned j numeral 255. Moreover, there may be many additionalchin variations that are depicted as 3-D facial feature parts andassigned a j numeral so that nearly all possible variations of thatfacial feature part are represented by one of the j-numbered parts. Forexample, just about every human chin will correspond to one of thej-numbered chins that are stored as a 3-D facial feature surface in thecomputer database. Similarly, facial feature shapes are stored in thedatabase for noses (FIG. 11), ears (FIG. 13) and all of the other facialfeature parts.

By separating each facial feature into a distinct 3-D facial featureparts, (e.g., mouth, ears, nose, eyes, chin for a total of 64 facialfeature parts) as shown in FIGS. 5 and 6, an inventory (repository) offacial feature parts and shapes is created. This repository is storedlocally in computer databases.

Any human face can be formed by selecting and assembling the facialfeature parts and shapes from the inventory, which is stored as thefacial feature databases. A law enforcement technician can create a 3-Dimage of the face of a suspect by selecting the shape version of eachfacial features parts that conforms to the suspects face. As shown inFIG. 32, the technician will select the nose shape, e.g., small nose,that looks most like the nose of the suspect. Similarly, the lawenforcement technician will select other facial feature shapes, e.g.,eyes, chin, mouth, etc., that look most like those features on the faceof the suspect.

When the technician has completed selecting the version of each facialfeature part that best corresponds to a suspect, the local lawenforcement computer assembles the selected facial feature parts togenerate a 3-D image of the suspect's face and head. The local lawenforcement computer 3206 has a local memory 3220 that contains adatabase of 3-D facial feature parts 3212. This database of facialfeature parts includes data which is representative of a 3-D surfaceimage of each of the shapes of all facial feature parts. For example,the facial feature parts database stored on each local computer memoryincludes a complete of all versions of facial feature parts and shapes.Thus, a 3-D image of any suspect's face and head can be generated fromthe facial feature parts database stored locally at each law enforcementorganization office.

The capability of storing locally a database from which a 3-D image ofany suspect's head allows local law enforcement to create a computergenerated image of a suspect using only the local law enforcementcomputer. The database of 3-D facial features occupies a relativelysmall amount of memory. The database 3212 of facial feature parts allowsthe law enforcement organization to generate 2-D images (such as frontaland profile views of suspects) by using a computerized facial imagegeneration system that stores composite codes and the facial featureparts to form a 3-D facial image. Moreover, a 2-D image of a suspect maybe generated from any point of view (or angle of view) because it isbased on a 3-D image stored in computer memory. The ability to generate2-D pictures of suspects from any angle or point of view is asubstantial improvement over existing systems that generate 2-D facialimages limited to straight-on frontal and profile views of suspects.

As shown in FIGS. 7 to 12 and 28 and 29, each facial feature part shapecan be referenced by a code, which consists of an “i” number and a “j”number. These codes form the composite code data 3214, and may be usedto identify mug shot photographs based on the facial features shown ineach photograph. The composite code data relates to a two-level indexingsystem for 3-D facial images. In particular, the composition code is atwo-layer hierarchical code in which a first (upper) layer (i code)relates to facial feature parts and the second (lower) layer (j codes)relates to the various facial surface shapes of each facial featurepart.

As shown in FIGS. 14 and 26, the “i” numbers represent a generic facialfeature part 1400, such as for example a chin (i=04), a nose (i=01), andears (i=14). The “j” number 1402 corresponds to a particular shape of ageneric facial feature part. The j numbers are assigned to each shapevariation of all other generic facial feature parts. In the disclosedexample, there are two-hundred and fifty six (256) variations(represented by j numbers) of each of the sixty-four (64) facial featureparts, represented by i numbers. The composite code 2602 is thecombination of i and j (part shapes 2604) numbers that are needed todefine a particular face. Each mug shot can be identified andreconstructed by a 64 Bytes composite code 2602 that identifies thespecific i and j numbers 2606 that correspond to the facial feature partshapes show in the mug shot.

The local law enforcement computer generates a 3-D image of the mug shotfrom the facial composite code data for the individual shown in the mugshot and the 3-D images of facial feature parts stored locally on thecomputer system. The flow chart in FIG. 34 shows the steps forgenerating a 3-D image of a human face. A composite code classificationprocess 3202, described below, determines the i and j numbers for aparticular face. Once the i and j numbers are determined for an entirecomposite code of a face, the code is correlated, step 3204, with thecomposite code database to match the composite code with the compositecodes of mug shots stored in database. The composite code database inthe local law enforcement computer stores the composite codes for alarge number of mug shots. The memory requirements for storing i, jcodes is substantially less than the memory that would be required tostore the images of all mug shots. Accordingly, it is more practical tostore locally just the i, j codes and not the mug shot images.

A comparative search through the local composite code database willquickly find a group of composite codes for mug shots that correspond tothe i, j codes used as the basis of the search, in step 805. The matchof i, j codes does not have to be exact, and the search may yield aconfidence rating for each match that indicates how close the j codesfor a particular mug shot are to the j codes used as the basis of thesearch. The i codes are always matched because they only refer to ageneric facial part, and all mug shot faces have all of the same genericfacial parts (unless some of the j codes are assigned to facial defectsor tattoos).

In step 3406, 3-D images of faces are generated from each of thecomposite codes that were located in the search of mug shot compositecodes in step 3405, and of the composite (i, j) codes that wereidentified in step 3404. The law enforcement computer generates a 3-Dimage by assembling as a mosaic of the 3-D surface facial feature shapesthat correspond to a particular j code for each of the facial featurecodes designated by the i codes. Accordingly, in step 3408, the localcomputer generates a 3-D image of a suspect from the composite (i, j)codes identified and/or the local computer generates a 3-D image of eachof the potential suspects whose mug shots match the composite codes thatwere used in the search. With the 3-D image, the local computergenerates a 2-D image for display on the monitor 3230 from anyperspective, e.g., profile, angled or frontal, that is desired. To viewdirectly the mug shot of any of the persons for whom a facial image hasbeen generated, the local law enforcement computer may access a local orremote mug shot photo database 3222 to obtain an actual photograph(s) ofthe individual and other information regarding the individual, in step3410.

The composite code classification process, step 3402, involves theassignment of codes to images of the faces individuals. In mostinstances, the composite code will be determined by mapping fiducialpoints on the 3-D facial image. As shown in FIGS. 9, 10, 28 and 29fiducial point 1000 is a particular spatial position on the surface ofthe face or head being imaged. A image surface can be defined in threedimensions by use of an appropriate number of fiducial points. In anembodiment of the present invention a relatively low-number (i.e., lowresolution) of fiducial points is used to uniquely identify each highresolution facial shape 1002 (j) in a group of a face part (i).

In step 3420, a facial image, (preferably a 3-D image), of a suspect isentered into the local law enforcement computer system. The image isanalyzed to determine the spatial dimensions of certain predeterminedfiducial points. FIGS. 9, 10, 28 and 29 show examples of 3-D fiducialpoints mapped on facial feature parts. The fiducial points are shown asan array of dots (see e.g., 1000) on parts for a chin, nose (FIG. 9),eyes, mouth and ears (FIGS. 10 and 29).

A relatively-small number of fiducial points, e.g., about 32 points, ismapped onto each facial feature part shape to form a unique fiducialtemplate 1006 for the shape, in steps 3422 and 3426. The fiducialtemplate is defined by the relative positions of the different fiducialpoints within the template. The fiducial points are located relative toother points in the template, and not to some other coordinate system.Because the fiducial points are located relative to one another it isrelatively straight-forward to compensate for the magnification and/orsize of the image when determining the fiducial points for a new facialimage. The fiducial points may be placed on their proper surfacepositions on the entire facial image and then the facial image withfiducial points is divided into facial feature parts. Alternatively, thefacial image may be divided into parts and then the facial features arepositioned on the parts of the image.

Moreover, it is possible to determine the fiducial points from two ormore 2-D images by placing the fiducial points on each image andallowing the computer system to transform the different 2-D positions ofeach point to a 3-D fiducial template. The law enforcement technicianplacing the points on the facial image must be trained in the properfiducial points to be placed on each facial feature part. Alternatively,the placement of fiducial points may be automated using standardcomputer imaging and recognition techniques. In addition, it is alsopossible to make a crude approximation of a 3-D fiducial template from asingle 2-D image, but the margin for errors and uncertainty isrelatively large for any template generated from the one 2-D picture.

Once a fiducial template has been create for an image of a facialfeature part (or for an entire facial image), then, in step 3426, thetemplate is compared to the templates of all the various facial shapesin the inventory (computer database 3212) for that facial feature part.A fiducial template is stored in memory (3220) for all shapes of eachfacial feature part. By comparing the fiducial template generated fromthe facial image, to the fiducial templates stored in memory, a matchcan be found of the facial image shape to one of the facial shapes inthe inventory of the computer. A law enforcement technician and thewitness may review the facial feature shapes that are found to match thefiducial template for the image and select the inventoried shape thatbest corresponds to the image and/or the recollection of the witness ofthe suspect's face.

When the best facial part shape from the computer repository 3212 hasbeen selected for each of the facial feature parts, then a image of thesuspect can be generated. In addition, the composite codes (i, j), forthe best matching facial feature shapes can be determined for use instoring data that can be used to redraw an image of the suspect and tomatch the suspect's facial image to the faces of persons having theirmug shots stored in a law enforcement mug shot data base. Accordingly,the present invention allows law enforcement organizations to utilize a3-D facial imaging system that allows for angled views, fast searchingof other facial images, indexing of 3-D facial features for futuresearches, and local computer storage of the data needed to redraw a 3-Dimage of any individual.

The invention has been described in connection with what is consideredto be the best mode. The invention is not limited to the disclosedembodiment, and covers various modifications and equivalent arrangementsincluded within the spirit and scope of the appended claims.

What is claimed is:
 1. A method for generating an image using a memoryunit, a processor and a display, wherein the method comprises thefollowing steps: (a) storing in the memory unit image feature partshapes, wherein the stored image feature part shapes arethree-dimensional surface images of image parts; (b) assigning a featurecode to each image feature part shape; (c) analyzing with the processora first image to determine the image feature part shapes selected frommemory that best match the first image; (d) identifying the featurecodes for the image features part shapes that match the first image; (e)storing in memory the feature codes for the first image that wasidentified in step (d); (f) generating, using the processor, a secondimage from a three-dimensional assembly of the image feature part shapesstored in step (a) that correspond to the feature codes identified forthe first image in step (d); (g) showing the second image generated instep (f) on the display; (h) storing in a memory unit a listing of imagecodes comprised of a set of feature codes where each image codecorresponds to a particular image; (i) matching the feature codesidentified in step (d) to at least one of the image codes stored in step(h); (j) generating a third image corresponding to the at least oneimage code matched in step (i) from an assembly of the image featureshapes, and (k) comparing the third image generated in step (j) to thesecond image generated in step (f).
 2. A method as in claim 1 whereinthe first image and the second image appear to be images of the sameobject.
 3. A method as in claim 2 wherein the object is a human face. 4.A method as in claim 3 wherein the image codes in step (h) eachcorrespond to a particular human face.
 5. A method as in claim 4 whereinsaid image feature part shapes include surface shapes of a plurality ofshapes of each of a nose, eye, ear and chin.
 6. A method as in claim 1wherein the first image is at least one two-dimensional image.
 7. Amethod as in claim 1 wherein the first image comprises a firsttwo-dimensional image, and a second two-dimensional image orthogonal tothe first two-dimensional image.
 8. A method as in claim 1 wherein thethird image is a three-dimensional image.
 9. A method as in claim 1wherein steps (h), (i), (j) and (k) are performed on a computer remotefrom the computer used to perform steps (c), (d), (e), (f) and (g). 10.A method for recognizing a facial image using at least one computer,said method comprising the steps of: (a) storing a three-dimensionalimages of facial part images and a facial feature code assigned to eachfacial part image; (b) analyzing a first facial image to match facialpart images to the first facial image; (c) identifying the facialfeature codes corresponding to the facial part images matching the firstimage; (d) storing the feature codes identified in step (c) as a set offeature codes representative of the first facial image; (e) generating athree-dimensional facial image from the set of feature codes stored instep (d), wherein the generated facial image is of a face visuallysimilar to a face shown in the first facial image; (f) storing a listingof sets of facial image codes, wherein each set of facial image codecomprises facial feature codes corresponding to a particular individual;(g) matching the facial feature codes identified in step (c) to thefacial feature codes of at least one of said sets of facial image codesin the listing, and (h) displaying a three-dimensional facial imagegenerated by an assembly of the facial image parts corresponding tofacial image codes of the at least one set of facial image codes matchedin step (g).
 11. A method as in claim 10 further comprising step (i) ofcomparing the facial image displayed in step (h) to the facial imagegenerated in step (e).
 12. A method as in claim 11 further comprisingthe step (j) of identifying an individual corresponding to the facialimage generated in step (e) based on the comparison in step (i).
 13. Amethod as in claim 10 wherein said facial part images include surfaceshapes of a plurality of shapes of each of a nose, eye, ear and chin.14. A method as in claim 10 wherein the first facial image is at leastone two-dimensional image.
 15. A method as in claim 10 wherein the firstfacial image comprises a first two-dimensional facial image, and asecond two-dimensional facial image orthogonal to the firsttwo-dimensional facial image.
 16. A method as in claim 10 wherein steps(f), (g) and (h) are performed on a computer remote from a computer usedto perform steps (c), (d) and (e).
 17. A method for generating a facialimage using a memory unit, a processor and a display, wherein the methodcomprises the following steps: (a) storing in the memory unit facialpart images, wherein the facial part images are three-dimensionalsurface images of facial features; (b) assigning a feature code to eachfacial part image; (c) analyzing, with the processor, a first facialimage to determine the facial part images matching the first image; (d)identifying the feature codes for the facial part images that match thefirst image; (e) storing in the memory unit the feature codes identifiedin step (d); (f) generating, using the processor, a second image from athree-dimensional assembly of the facial part images stored in step (a)that correspond to the feature codes identified in step (d); (g)displaying the second image generated in step (f); (h) storing facialpart images and their corresponding feature codes for a plurality offaces; (i) matching the feature codes identified in step (d) to thefeature codes stored in step (h), and (j) identifying one of the facesfrom step (h) based on the match performed in step (i).